Papers with sampling-based method
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values (2023.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LMs) have been shown to be highly effective for identifying harmful training instances, but dataset size and model complexity constraints limit the ability to apply Shapley-based data valuation to fine-tuning large pre-trained language models. |
| Approach: | They propose an algorithm that aggregates Shapley values from subsets for valuation of entire training set and a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets and can filter fine-tuning data to increase language model performance compared to training with the full fine-uning dataset. |
Quantifying Similarity between Relations with Fact Distribution (P19-1)
Copied to clipboard
| Challenge: | a conceptually simple and effective method to quantify the similarity between relations is presented . identifying relations is a crucial problem for several information extraction tasks. |
| Approach: | They propose a method to quantify the similarity between relations in knowledge bases . they use a neural network to parameterize conditional probability distributions over entity pairs . |
| Outcome: | The proposed method significantly correlates with human judgments, the authors show . it could be incorporated into negative sampling and softmax classification to alleviate these mistakes. |
SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection rely on self-consistency check alone . prominent LMs exhibit a tendency to produce exceedingly confident, but erroneous, assertions . |
| Approach: | They propose a sampling-based method that expands on the principle of self-consistency checking to detect hallucinations at question-level and model-level. |
| Outcome: | The proposed method outperforms the state of the art in detecting non-factual and factual statements across multiple question-answering and open-domain generation benchmarks. |